On the Kozachenko-leonenko Entropy Estimator
نویسنده
چکیده
We study in details the bias and variance of the entropy estimator proposed by Kozachenko and Leonenko [10] for a large class of densities on Rd. We then use the work of Bickel and Breiman [2] to prove a central limit theorem in dimensions 1 and 2. In higher dimensions, we provide a development of the bias in terms of powers of N−2/d. This allows us to use a Richardson extrapolation to build, in any dimension, an estimator satisfying a central limit theorem and for which we can give some some explicit (asymptotic) confidence intervals.
منابع مشابه
Effectiveness of the Kozachenko-Leonenko estimator for generalized entropic forms.
In this Brief Report we discuss the effectiveness of the Kozachenko-Leonenko entropy estimator when generalized to cope with entropic forms customarily applied to study systems evincing asymptotic scale invariance and dependence (either of linear or nonlinear kind). We show that when the variables are independently and identically distributed the estimator is only valuable along the whole domai...
متن کاملAnalysis of k-Nearest Neighbor Distances with Application to Entropy Estimation
Estimating entropy and mutual information consistently is important for many machine learning applications. The Kozachenko-Leonenko (KL) estimator (Kozachenko & Leonenko, 1987) is a widely used nonparametric estimator for the entropy of multivariate continuous random variables, as well as the basis of the mutual information estimator of Kraskov et al. (2004), perhaps the most widely used estima...
متن کاملEfficient multivariate entropy estimation via k-nearest neighbour distances
Many statistical procedures, including goodness-of-fit tests and methods for independent component analysis, rely critically on the estimation of the entropy of a distribution. In this paper, we seek entropy estimators that are efficient in the sense of achieving the local asymptotic minimax lower bound. To this end, we initially study a generalisation of the estimator originally proposed by Ko...
متن کاملOn the effectiveness of a binless entropy estimator for generalised entropic forms
In this manuscript we discuss the effectiveness of the Kozachenko-Leonenko entropy estimator when generalised to cope with entropic forms customarily applied to study systems evincing asymptotic scale invariance and dependence (either linear or non-linear type). We show that when the variables are independently and identically distributed the estimator is only valuable along the whole domain if...
متن کاملThe Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal
We analyze the Kozachenko–Leonenko (KL) nearest neighbor estimator for the differential entropy. We obtain the first uniform upper bound on its performance over Hölder balls on a torus without assuming any conditions on how close the density could be from zero. Accompanying a new minimax lower bound over the Hölder ball, we show that the KL estimator is achieving the minimax rates up to logarit...
متن کامل